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Abstract 

A theoretic framework for multimedia information retrieval is intro- 
duced which guarantees optimal retrieval effectiveness. In particular, a 
, Ranking Principle for Distributed Multimedia-Documents (RPDM) is de- 

^J ■ scribed together with an algorithm that satisfies this principle. Finally, 

rA. ' the RPDM is shown to be a generalization of the Probability Ranking 

Q_) , principle (PRP) which guarantees optimal retrieval effectiveness in the 

case of text document retrieval. The PRP justifies theoretically the rel- 
evance ranking adopted by modern search engines. In contrast to the 
►-^ . classical PRP, the new RPDM takes into account transmission and in- 

f^ ' spection time, and most importantly, aspectual recall rather than simple 

r\i . recall. 

o 

, 1 Introduction 

0\ 

iz^ , Multimedia Information Retrieval is becoming more and more feasible because 

Y^ • both speech and image recognition methods have been improved significantly 

during the last years: A wealth of new information access techniques have 
been developed to find relevant information in large multimedia data collections 

K> , ( ^chaublc, 1997 ). While some of these ne w techniques - f or example retrieval 



techniques for digitized speech documents ( Wechsler, 1998| ) - have been evalu- 



b. - . - - ' — - 

5t 1 ated experimentally, they were hardly developed and studied within a theoretic 

framework that guarantees optimal retrieval effectiveness; in fact we lack such a 
framework that optimizes the probability that a user finds the desired informa- 
tion in a large multimedia document collection. In this paper, a Ranking Princi- 
ple for Distributed Multimedia (RPDM) document collections is described that 
serves as a theoretic framework like the Probability Ranking Principle (PRP) 
by Robertson (1977) for centrally stored text documents. The PRP states that 
a retrieval system performs optimally if the documents are ranked according to 
decreasing probabilities of relevance. He showed that optimal performance can 
be expressed either in terms of precision and recall, or in terms of costs as- 
sociated with the retrieval of non-relevant documents and the non-retrieval of 
relevant documents. We elaborate on the PRP in Section l5l 



Nowadays, large distributed multimedia document collections are available. 
Local and global computer networks, for example the World-Wide Web, allow 
quick access and transfer of documents independent of their location. Further, 
new technologies enable digital processing of non-text media, such as images, 
audio and video, which collectively emboss the notion of m,ultim,edia. 

2 Criteria for Distributed Multimedia Documents 

For IR systems managing distributed multimedia documents, we believe that 
the optimal document ranking problem has to be revised. Such IR systems 
should allow for additional criteria other than solely the probability of relevance 
when suggesting an inspection order for documents with respect to a query. We 
identify the following two additional criteria: 

1. Transmission time of a document, the time needed to transport a docu- 
ment from the source location across the network to the user. 

2. Inspection time of a document, the time needed by the user to inspect a 
document. 

We demonstrate the importance of these criteria by the following two exam- 
ples that are also illustrated in Figure IT]: (1) Assume that two documents dj 
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Figure 1 : Retrieval of distributed multimedia documents and parameters affect- 
ing optimal ranking. 



(text) and d^ (text) have an equal probability of relevance to a given request. 
If dj is geographically closer to the user than dfc, then obviously dj should be 
transmitted and presented before d^, such that the user's idle waiting time is 
minimized. (2) Assume that two documents d^ and di contain the same infor- 
mation but are from different media, say d^ is text and di is audio or video. 



In this case the IR system should favor the ranking of the text document first, 
since it is much faster both to transmit and to inspect. 

The transmission time is affected (1) by the geographical distance, (2) by 
the network bandwidth, and (3) by the document's storage size, which depends 
on the document's medium and length. The media text, audio and video are 
orders of magnitude apart from each other with regard to their associated data 
rates. Table [^ shows t he data rates for different (un)compressed formats of the 
three media (Lu, 1996, p. 49, 108): The text data-rate is based on a speaking rate 



document medium 


data format 


data rate 


text 


ASCII 


0.018 kByte/s 


audio 


uncompressed CD-audio 


176.4 kByte/s 


audio 


compressed CD-audio (MPEG-audio) 


29.4 kByte/s 


video 


uncompressed VHS-video 


6750 kByte/s 


video 


compressed VHS-video (MPEG-1) 


187.5 kByte/s 



Table 1: Data rates for different compressed and uncompressed media. 



of 120 words per minute and an average word-length of 9 letters. For audio and 
video, sophisticated compression techniques such as MPEG ( Pan, 199^ ) make 
a data reduction possible, however the differences between the media are still 
considerable. 

The inspection time is dependent on the medium and the length of the 
document as well. Due to the time-synchronous nature of audio and video, 
these media require much more time for inspection compared with for example 
an equivalent text document. Also, it has been found experimentally that users 
requi re more time to extr act requested information from a video than from a 
text ( ISutchffe et al., 1997| ). 



3 Ranking Principle 

Before we study the optimal ranking problem in an IR system, we make two 
general assumptions: First, we assume that the IR system is able to return 
(query-dependent) passages rather than documents in response to a query. Pas- 
sages are motivated mainly by the fact that entire audio and video documents 
require very much time for transmission and inspection, and by the fact that 
a user would like to listen only to relevant parts of a (maybe long) recording. 
The second assumption is that a query q consists of k aspects qi,i = l,...,fc. 
For example, the query "W.A. Mozart" may ask for information about Mozart's 
compositions but also about his life in society. 

To study the optimal ranking problem for distributed multimedia documents 
we assume that a user specifies a total inspection time when he submits a query. 
We further assume that, after query evaluation, the IR system has the following 
parameters available for each passage: (1) the probability of relevance, which 



is estimated by any retrieval method, (2) the transmission time, and (3) the 
inspection time. 



Ranking Principle for 

Distributed Multimedia Documents (RPDM) 

An IR system should present passages of distributed 
multimedia documents to a user query in such a way 
that 

1. The passages can be inspected within the user- 
specified total inspection time, 

2. There is no user waiting time between the inspec- 
tion of passages due to their transmission, 

3. The passages contain "a maximum amount of rel- 
evant information about various aspects of the 
query" . 



More formally, we formulate the RPDM as an optimization problem. Given 
are the 

• total inspection time T specified by the user, 

• the set X of passages Xj,i = 1, ...,\X\ found to query g, 

• transmission time i* {xj ) for each passage Xj , 

• inspection time t* {xj ) for each passage Xj , 

• probabilities of relevance P{R\qi,Xj) for each passage Xj with regard to a 
query aspect q^. 

The problem for the IR system is to find a sequence Y — (j/i , 
j/j e X such that the following conditions are satisfied: 



.,j/„} of passages 
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(1) 

(2) 
(3) 



The first condition ffl) requires that all selected passages can be transmitted 
and inspected within the total inspection time T. 

The second condition (H) assures that the next passage to be inspected can 
be transmitted completely within the time used for the transmission of the first 
passage and the inspection of all preceding passages. Here we assume that 
the IR system is capable of requesting future passages in the background while 



the user is inspecting. This ehminates waiting time due to the transmission of 
passages. Figure illustrates two time scenarios of the user's inspection process, 
where condition (0) is satisfied only in scenario (a). 

Definition is the main cost function which has to be minimized. We justify 
the choice for the cost function as follows: The term (1 — P{R\qi^ y)) denotes the 
probability that passage y does not cover aspect qt , and thus the product denotes 
the probability that aspect qi is not at all covered in the selected passages. If 
we associate constant costs for each aspect not covered, the cost function C(Y) 
is proportional to the expected costs for missing relevant aspects of the query. 
In other words, the IR system should suggest passages that optimally cover the 
most aspects to the given query. Our cost function is inversely proportional 



to th e aspectual recall used in TREC's interactive track ( Vorhees fc Harman, 



1999). In the next section we present an algorithm that solves this optimization 



problem. 

4 An Algorithm Satisfying the RPDM 

The simplest algorithm to the problem formulated in the previous section is a 
backtracking procedure ([Nicvergelt, 1977)(Kreher fc Stinson, 199^). The idea of 



backtracking applied to our situation is to enumerate all sequences of passages 
(i.e. possible solutions) while (1.) checking both conditions dll) and (0) for 
each sequence, (2.) computing its cost function C(Y), and (3.) retaining the 
sequence that obtains minimal costs. 

We show, however, that backtracking is not feasible for this problem since 
the number of solutions to be considered is far too high. Let TV = \X\ be the 
total number of possible passages that have been found to a given query, and 
assume that the algorithm selects n passages. The number of solutions with n 
passages can be derived by combining n out of N passages and by permutating 
those sequences, which results in 

iV\ I _ N\ 



n) ' {N 



W 



solutions. Since n may vary from 1 to A^, the total number of solutions to be 
considered is 

For TV = 10 this value is « lO"^ and for A^ = 100 it is Ri 2 • W^^ . 

To reduce the number of solutions to consider, we propose the classic branch- 



and-bound algorithm (e.g. (Domschkc & Drexl, 1991, p. 114-119)), which is a 



variant of the backtracking algorithm. The idea of branch-and-bound is (1) 
to enumerate solutions with lowest expected cost-values first (branch), and (2) 
to discard entire sets of solutions where costs can certainly not be further re- 
duced (bound). The pseudo-code of the algorithm is given in Figure 0. The 
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Figure 2: User inspection process for three passages. Future passages are trans- 
mitted in the background, (a) No user waiting time, (b) user waiting time 
between the first and second passage. Note also that in (b), the last passage 
cannot be inspected completely. 





VAR 




Ybest', /* currently best solution */ 




Cbest': /* costs of best solution */ 


1 


PROCEDURE brancli.and.bound( Sequence Y ) 


2 


VAR 


3 


Cy; /* costs for solution Y */ 


4 


LBC{Y^); /* lower bound costs for extensions of {Y,x) */ 


5 
6 

7 


Ey; /* set of possible passage extensions of Y */ 


Cy := CiY); 


8 


IF Cy < Chest 


9 


Ybest '■— Y] Chest '■— Cy 


10 


END; 


11 


FOR ALL xeX\ set(y) 


12 


IF within_user_time{{Y, x)) A no_waiting{{Y, x)) 


13 


calculate LBC(K+); 


14 


Ey := Ey U {x}; 


15 


END 


16 


END; 


17 


LOOP 


18 


X := argmin{LBC(y+) \ x e Ey A LBC(r+) < Cbest} 


19 


IF X undefined 


20 


BREAK; 


21 


END 


22 


branch_and_bound{{Y, x)); 


23 


END; 


24 


RETURN 


25 


END PROCEDURE; 




/* INPUT: */ 




/* X: set of passages found to a request */ 




/* i*(x): transmission times */ 




/* i*(x): inspection times */ 




/* P{R\qi,x): probabilities of relevance */ 




/* T: total user inspection time */ 




MAIN 




Ybest := e; Cbest := oo; 




branch-and-bound{e) ; 




END 




/* OUTPUT: Ybest */ 



Figure 3: Recursive branch-and-bound algorithm satisfying the RPDM. 



algorithm is shown as a recursive procedure branch_and-bound that calculates 
the cost function for a given sequence Y and tries to extend this sequence with 
an additional passage at the end, before it is called recursively. The main call 
of this procedure is branch_and-bound(e), where e denotes the empty sequence. 
Figure ^ illustrates the extension of the sequence Y graphically. The circles 
correspond to solutions and the arrows denote extensions with a single passage. 



LBC(Y^^ ) 



LBC(Y^) 




Figure 4: Search tree of the branch-and-bound algorithm. Each node represents 
a solution. Successor nodes are extensions of the solution Y. The LBC-values 
are used either to select an extension as the next solution, or to discard an entire 
subtree. 



In line 7 of the algorithm, the costs Cy of the current solution are calculated 
according to the cost function (Definition ||). Then, the currently best solution 
Ybest is updated if necessary. The rest of the procedure considers extensions of 
Y with an additional passage x. We write {Y,x) for a new sequence consisting 
of Y and an appended passage x. Also, we define Y^ as the set of all sequences 
that consist of the sequence Y, an appended passage x and any number of fur- 
ther passages. In lines 11-16, for each extension passage x and each resulting 
set Y^, a lower bound of costs LBC{Y^) is calculated (line 13, see next para- 
graph), and the extension passage is retained in an extension set Ey (line 14). 
However, this is only performed if two conditions are met (line 12): The proce- 
dure within_user_time(Y,x) tests the condition (|l|), and no-waitingfYjx) tests 
the condition (0). Finally, in lines 17-23 the extension with the lowest LBC- 



value is pursued by a reeursive eall of branch-and_bound (branch). In line 18, 
the best passage is selected for extension, and simultaneously, extensions are 
only pursued if their costs (represented by the lower bound costs) may become 
smaller than the costs of the currently best solution Cbest (bound) . 

In the following, we derive a lower bound of costs LBC(Kj'") for all solutions 
consisting of the solution Y extended with a passage x and any number of 
further passages: According to the cost function (Definition 0) we write for the 
costs of solution Y 

C{Y) := \{{l-P{R\q,,y)) + ...+ \{{l^P{R\qk.,y)) 

yeY yeY 

'^ V ' " v ' 

P{qi) P(qk) 

where P{qi) abbreviates a product term. If we extend the solution Y with a 
passage x, it holds that 

C{{Y,x)) := P{qi)-{l-P{R\qi,x)) + ... + P{qk)-{l~P{R\qk.x)) 

> CiY)- min {l-P{R\q,,x)}. (4) 

i—l,...,k 

Now we consider any solution Y' £ Y^ , which starts with the sequence Y 
followed by the passage x and any number of further passages. From Equation |j 
follows: 

C(Y') > C(Y)- mm {l~P{R\q„x)}- 

i—l...k 

{mm{l -P{R\qi,x')\i = l,...,kAx' e X \ set{{Y, x))})"\ (5) 
" V ' 

= : r{Y,x) 

where r{Y, x) is an abbreviation and m denotes the maximum number of pas- 
sages that may be added to the sequence (Y, x) . This number is limited by the 
remaining inspection time and by the minimum inspection time of the passages 
not yet selected. Let ty be the time necessary to inspect all passages of the 
solution Y . Then we can formulate an upper bound for m by 



< 



T -ty - t\x) 



mm{t^{x')\x' (zX\set{{Y,x))} 



Thus, the right part of Equation a contains a lower bound for the costs of a 
solution Y' e Y^ and therefore we can define 

LBC{Y+) := C{Y)- min {l-P{R\q„x)}-r{Y,x). 

i—l...k 

The lower bound costs denote the minimum possible costs of any solution ex- 
tended from a solution Y and the passage x. If they are larger than the cur- 
rently minimum costs, then the entire search subtree {Y,x) (Figure 0) may be 
discarded. 



The output of the branch-and-bound algorithm satisfies the RPDM as stated 
on page 0. A crucial element of the branch-and-bound algorithm is the deriva- 
tion of a highest possible lower bound for the costs of extended solutions. The 
higher the lower bound is, the more efficient the algorithm is because the num- 
ber of solutions to consider is reduced. However, more accurate lower bounds 
are usually more complex to compute. This results in a trade-off between the 
effort to compute a lower bound and the effort to consider more solutions. 

In a practical situation it may be that the IR system does not have enough 
time to run the branch-and-bound algorithm because the search tree is still too 
large. For such cases we propose the use of suboptim al stochastic optimiza- 
t ion techniques (e.g. simulated annealing ( Vidal, 1993| ) or genetic algorithms 



( Mitchell, 1996| )). These approaches do not guarantee that the best solution 



will be found, but they may find a sufficiently good solution in a short time. 

5 The RPDM and Robertson's PRP 

In this section we show that our ranking principle for distributed multimedia 
collections is compatible to Robertson's Probability Ranking Principle (PRP) 
which was developed in the context of retrieval in rather short texts and bibli- 



ographic records (Robertson, 1977) 



We map our distributed multimedia model into the context of the PRP, 
which originally was to retrieve bibliographic references or (rather short) text 
documents. We make the following assumptions: 

• We do not deal with passages but with documents because the (text) 
documents are very short (less than 1 page). 

• The transmission time of each document is neglected {t*{dj) — Vj) be- 
cause each document consists of only a few text words. 

• The inspection time t^{dj) for each document is constant, say At, because 
the short text documents are able to be inspected at a glance. 

• There is only one aspect in the query. 

Again, we assume that the user specifics a total inspection time T. With 
constant inspection times, it follows that the user inspects n := [xtJ documents. 
Let Y be the sequence of those n documents. Since the transmission times are 
neglected, the documents of Y may be presented in any order. 

The cost function (Definition 0) is simplified due to the presence of only one 
aspect: 

C{Y) := l[il-PiR\q,d,)). (6) 

djeY 

Proposition: If the solution Y contains the top n documents with respect 
to P{R\q,dj),dj e Y (and thus conforms to the PRP), it follows that its costs 
C{Y) are minimal. 



10 



Proof (by contradiction): Assume that the solution Y contains the top 
n documents, and that its costs are not minimal. We can choose any document 
d G y and replace it by any other document d! ^ Y from the rest of the 
collection. According to Definition the costs of this new set Y' are 



l-P{R\q,d') 
l-P{R\q,d) 



dj£Y 

■C{Y). 



■■f 



Since P{R\q,d) > P{R\q,d'), it follows / > 1 and thus C{Y') > C{Y) or C{Y) 
is minimal, q.e.d. 

The statement made in the proposition conforms to the Probability Ranking 
Principle under the assumption that the number of documents to inspect is 
known a priori. Thus we have shown that the context of the Probability Ranking 
Principle is a special case of our RPDM, and that the PRP is satisfied if the 
RPDM is satisfied. 

6 Conclusions 

We have introduced a new ranking principle for distributed multimedia doc- 
uments (RPDM). This principle describes the optimal ranking of distributed 
multimedia documents by taking into account not only the documents' proba- 
bilities of relevance, but also their transmission and inspection times. We have 
formulated the ranking principle as an optimization problem, and we have pre- 
sented an algorithm with which the principle can be satisfied. Lastly, we have 
shown that the RPDM is a generalization of Robertson's probability ranking 
principle. 
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